Multi-screen perspective altering display system

ABSTRACT

The perception of a displayed image is altered for viewers moving relative to a display screen, thereby imparting a sense of three-dimensional immersion in the scene being displayed. A display generator generates a scene having foreground and background elements, and a display screen displaying the scene. A sensor detects the position of a viewer relative to the display screen, and a processor is operative to shift the relative position of the foreground and background elements in the displayed scene as a function of viewer position, such that the viewer&#39;s perspective of the scene changes as the viewer moves relative to the display screen. The foreground and background elements may be presented in the form of multiple superimposed graphics planes, and/or a camera may be used to record the scene through panning at sequential angles. The system may be used to implement virtual windows, virtual mirrors and other effects.

REFERENCE TO RELATED APPLICATION

This application is a Divisional of U.S. patent application Ser. No.16/046,065, which claims priority and the benefit of U.S. patentapplication Ser. No. 12/197,635, filed Aug. 25, 2008, which claimspriority and the benefit of U.S. Provisional Patent Application Ser. No.60/957,845, filed Aug. 24, 2007, the entire content of all applicationsbeing incorporated herein by reference.

FIELD OF THE INVENTION

This invention relates generally to display systems and, moreparticularly, to systems that alter perspective, synthesize depthperception and provide other capabilities, thereby enhancing the viewingexperience.

BACKGROUND OF THE INVENTION

Flat panel displays are growing in size and falling in price. At thistime, non-projection, true HD (i.e.—1080 p) 50″ displays are availablein the $1,000 to $2,000 price range, and new models are introduced on aregular basis. When edge-lit liquid-crystal display (LCD) panels arereplaced with back-lit white light-emitting diodes, another leap intechnology will occur. When organic LED panels become viable, flexible,affordable wall-sized displays, including wrap-around configurations,should be possible.

As large displays proliferate, users are finding uses for them beyondjust “watching TV.” In some cases, for example, users are displayingpictures or rotating sequences of pictures on these displays, therebycreating, in effect, large picture frames. However, existing systems forvideo display are restricted either in their utility or in theirrealism, due to the inherent limitations in a two-dimensionalpresentation unit. Of these, the most important impact to the perceptionof a scene may be its lack of visual perspective as the viewer changeshis position relative to the display unit.

The use of 3-D glasses or other paraphernalia is oppressive in thosecircumstances where a casual effect is desired, or where no userintervention is to be required. Thus, prior-art systems relying on stillor even video images, or various photographic or video projectiontechniques, cannot achieve the level of perception of reality that isdesired for many residential or commercial display applications.

SUMMARY OF THE INVENTION

This invention resides in apparatus and methods providing a uniqueexperience for the viewer of a display, particularly large wall-mountedpanels. In the preferred embodiments, the perception of a displayedimage is altered for viewers moving relative to the position of thedisplay system screen, thereby imparting a sense of three-dimensionalimmersion in the scene being displayed.

A perspective-altering display system according to the inventioncomprises a display generator for generating a scene having foregroundand background elements, and a display screen displaying the scene. Asensor detects the position of a viewer relative to the display screen,and a processor is operative to shift the relative position of theforeground and background elements in the displayed scene as a functionof viewer position, such that the viewer's perspective of the scenechanges as the viewer moves relative to the display screen.

The foreground and background elements may be presented in the form ofmultiple superimposed graphics planes. At least one of the graphicsplanes may include prerecorded material or material received through atransmission medium or camera. A camera may be used to record the scenethrough panning at sequential angles, with a memory being used to storethe images obtained at the sequential angles for later recall as afunction of user movement. An interpolator may be used to “fill in”visual gaps in the scene.

The display screen may be mounted on a wall having a backside, and acamera may be mounted on the backside of the wall which pans as a viewermoves, thereby imaging a scene representative of the display being avirtual window through the wall. A camera with a zoom capability may beused for recording the scene. With the sensor being operative to detectthe viewer's distance from the display screen, the processor is furtheroperative to zoom in the camera as the viewer moves closer to thedisplay screen, and zoom out the camera as the viewer moves away fromthe display screen.

The sensor may be operative to detect the viewer's distance from thedisplay screen, with the processor being operative to increase theresolution of the scene as the viewer moves closer to the display screenand decrease the resolution of the scene as the viewer moves away fromthe display screen. A camera with a tilt capability may be used forrecording the scene, with the processor being further operative to tiltthe camera in response to the viewer's up/down movement. The camera mayhave a field of view which includes a viewer of the display screen,enabling the display to function a virtual mirror.

A plurality of cameras may be used for capturing the scene, with theprocessor being further operative to construct a three-dimensional imagefor display on the screen. A user control may be provided enabling aviewer to select a specific camera or cameras to see how others wouldview the user from different perspectives.

The sensor may include an infrared CCD (charge-coupled device) camera.Regardless, the camera may have an image sensor, such that an image of aperson in front of the display may be focused onto the image sensor as aspot or group of pixels, allowing the movements of the person to betracked with no moving parts.

The sensor may be a camera having at least a pan mount that tracks themovement of a viewer, with the processor being operative to shift therelative position of the foreground and background elements as afunction of the tracking. The sensor may be a camera having at least apan/tilt mount that tracks the movement of a viewer, with the processorbeing operative to shift the relative position of the foreground andbackground elements as a function of the tracking. The sensor may be acamera having an auto-focus capability.

The sensor may be operative to sense a plurality of individuals in frontof the display screen. The processor may be operative to shift therelative position of the foreground and background elements by favoringlarger clusters of individuals as opposed to smaller clusters or singleindividuals. Alternatively, moving individuals may be favored overstationary individuals, or individuals actually looking at the displayscreen may be favored over those who are not.

One or more transducers may be provided for producing sounds associatedwith the scene, with the processor operative to alter the reproductionof the sounds as a function of viewer movement.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A shows a viewer having a field of view walking from the righttowards the left relative to a display screen (shown as viewed fromabove);

FIG. 1B is a simplified representation of what the person might see onthe display screen according to the invention from the position shown inFIG. 1A;

FIG. 2A shows the person moving to the left, closer to the center of thedisplay screen;

FIG. 2B shows how closer objects have shifted laterally to the greatestdegree, as opposed to more distant objects, much as a train passengerwould experience while looking out the window of the train as the trainmoves;

FIG. 3A illustrates how the viewer has moved to the left-most portion ofthe display;

FIG. 3B shows how close objects have shifted to the extent that they nowpartially overlap with an object further away;

FIG. 4A depicts the image of a foreground object being gathered by acamera and recorded by a recorder;

FIG. 4B shows mid-range objects being recorded;

FIG. 4C shows distant or background objects are being recorded;

FIG. 5 depicts an alternative technique for implementing perspectivealteration according to the invention;

FIG. 6A shows how an infrared CCD (charge-coupled device) camera,preferably with a wide-angle lens, may be used as a sensor according tothe invention;

FIG. 6B shows the use of a panning camera;

FIG. 6C depicts three persons generating a composite thermal field on animage sensor;

FIG. 7A illustrates a “virtual picture window” embodiment, wherein themovement of a viewer causes an outdoor camera to pan back/forth, therebyallowing the viewer to visualize the outdoor scene as if the displaywere a hole in the wall;

FIG. 7B shows how the invention enables virtual windows on insideswalls, which may be useful in homes and businesses such as restaurants,bars and nightclubs;

FIG. 7C shows how the display can function as a “virtual mirror”; and

FIG. 8 depicts how the virtual mirror embodiment of the invention may beused in bathrooms and dressing rooms.

DETAILED DESCRIPTION OF THE INVENTION

This invention employs a variety of techniques to provide a uniqueexperience for the viewer of a display, particularly large wall-mountedpanels. In the preferred embodiments, the perception of a displayedimage is altered for viewers moving relative to the position of thedisplay system screen, thereby imparting a sense of three-dimensionalimmersion in the scene being displayed.

FIG. 1A shows a viewer 102 having a field of view 104 walking from theright towards the left relative to a display screen 106. One or moresensors 108, which may use visible-light, infrared, ultrasonic, or othermodalities described in further detail below, are used to track at leastthe lateral position of individual 102.

FIG. 1B is a simplified representation of what the person 102 might seeon the display screen 106 according to the invention from the positionshown in FIG. 1A. Relatively close objects are shown at 115, 116. Lessclose objects are seen at 114. Somewhat distant objects are shown at112, and distant objects are shown at 110. Although four relativedistances are mentioned, the invention is not limited in this regard,and is applicable to more or fewer such relative distances.

In FIG. 2A, the person 102 has moved to the left, closer to the centerof the screen 106. The sensor(s) 108 have detected this movement and, inresponse, the perspective of the scene has been altered. As shown inFIG. 2B, closer objects have shifted laterally to the greatest degree,followed by objects 114 and 112 in order, much as a train passengerwould experience while looking out the window of the train as the trainmoves. Object 110, being significantly distant, would shift little, ifat all.

This process continues in FIG. 3A, where the viewer has moved to theleft-most portion of the display. Close object 116 has shifted to theextent that it now partially overlaps with an object further away, andobject 115 has begun to move off the screen 106, as shown in FIG. 3B.The ways in which the invention makes this possible will now bedescribed in further detail.

In the preferred embodiment, the invention employs a technique similarto that utilized in animation films: multiple superimposed graphicsplanes. In FIG. 4A, the image of a foreground object 412 is gathered bycamera 402 and recorded by recorder 410. Depending upon thecircumstances, a blue or other solid-color background 414 may be usedfor chroma-keying.

In FIG. 4B, mid range objects 420 are being recorded whereas, in FIG.4C, distant or background objects are being recorded. Commerciallyavailable software packages, or customized software tuned to specificprogram content, can be utilized to derive the desired material fromamong multiple subjects representing different focal points, and, onceidentified, can track the subjects as they change their position andeven their orientation. In an alternative approach, multiple cameras,positioned to capture three-dimensional information, may be utilized toderive a three-dimensional array for each frame of motion, therebyallowing the producer to select “slices” which can be captured asgraphics plane images for manipulation by the graphics processor of theinstant invention.

The camera(s) may record moving video images for the foreground graphicsplane(s), the background graphics plane(s), or any combination thereof.For example, the background graphic plane may be based upon a stillpicture, while the foreground cameras record motion imagery. In thisembodiment, position sensor(s) 108 detect the location of the viewerrelative to the screen, and reposition the foreground graphics plane(s)as the viewer moves, thereby conveying to the viewer the impression, forexample, that he is looking through a window at an outdoor scene, with,perhaps, a nearer image, such as a tree branch, that the viewer can seearound by simply shifting his position relative to the display screen.The tree branch (or other object) may also be moving, as it would in abreeze, for example.

The video source for these graphics planes may include prerecordedmaterial supplied by playback from any recording devices. Other sourcesinclude broadcast, satellite, cable, or other programming sources,material delivered over broadband or other telecommunication links,privately recorded material, live video from cameras (including securityand monitoring cameras), computer-generated graphics and the like, orany other source of image material. Graphics planes displaying textinformation may be superimposed over, or under, other graphics planes.

FIG. 5 depicts an alternative technique for implementing perspectivealteration according to the invention. Here a camera 502 having at leasta pan mount 506 records an actual scene 512 at incremental anglessuggested by arrow 508. The view at each angle is recorded by unit 510for later replay. The number of increments depends upon the desiredresolution, room dimensions, and other factors. For example, at a veryhigh resolution, single-degree increments may be recorded through a full180 degrees. At a lesser resolution, single-degree increments may berecorded across a smaller angle of view, or larger-degree increments maybe used at angle of view with or without interpolation to fill in any‘gaps’ during replay.

Thus, a variety of novel video applications are enabled by virtue of theinvention, such as a simulated window which has a view that is notpossible from that position if there were an actual window in thatposition (i.e., viewing a sunset from an eastern exposure), or a viewwhich is not possible at all (i.e., a winter scene during the summer, ora scene from a different country). These represent a clear improvementover any of the typical “light box” with photographic transparency orother attempt to simulate a window view with conventional means.

FIG. 7A illustrates a “virtual picture window” embodiment, wherein themovement of viewer 102, detected by sensors 108, cause outdoor camera704 on mount 706 to pan back/forth, thereby allowing the viewer 102 to“see” the outdoor scene 720 “through” the wall 702 on display 106 usinggraphics processing system 710. Depending upon the movement of theviewer toward/away from the screen 106, in this and other applicableembodiments the camera 704 may zoom as the viewer comes closer, pan whenthe user moves away and tilt if the viewer goes up/down, as might be thecase on a staircase, for example.

Through the use of distance detection, the display system can provide athree-dimensional effect, by applying modifications to an image as theviewer changes his position. Thus, for example, an image of a paintingmight capture the artist's intentions when viewed at a distance, while aclose-up examination of the video display by the viewer could displaythe details of the brush strokes as the viewer changed his positionrelative to the screen.

The invention is not limited to virtual windows through outside walls.As shown in FIG. 7B, the invention enables virtual windows on insideswalls, which may be useful in homes and businesses such as restaurants,bars and nightclubs. Similar to the embodiment depicted in FIG. 7A, asuser 102 moves relative to screen 106, camera 704 pans (or tilts orzooms), enabling the user 102 to see people 730 or other objects throughthe wall 702.

In still another aspect, the display system can function as a “virtualmirror.” Such an embodiment is depicted in FIG. 7C, wherein graphicsprocessing system 710 directs the camera 704 to at least pan at an angle“A” substantially equal to angle “B” formed by the location of the userand line 722 perpendicular to the plane of display 106. This allows theviewer to see a synthesized reflection depicted by broken line 720,typically including the subject him/herself.

The mirror embodiment of the invention may be used in bathrooms anddressing rooms, as illustrated in FIG. 8. Multiple cameras 802, 804,806, 808 disposed around the display screen 106 allow the processor unit812 to construct a three-dimensional image for display on the screen,and the proximity sensing devices may be utilized to create an on-screenimage which is representative of what a person should see as here-positions himself, in three dimensions, about the display screen. Itis a simple matter to provide a left-to-right reversed-image, in keepingwith the mirror aspect of the invention. A user control 810 allows theuser to select a specific camera or cameras to see how others would viewthem from different perspectives.

In the dressing room/bathroom embodiments of the invention, thecamera(s) capturing the image of the viewer may be placed behind asemi-transparent screen allowing better visualization such as eye-levelcontact to be maintained. One or more video cameras may be provided on abendable tether—or wireless hookup—enabling a user to view hard-to-reachplaces such as ears, nose, mouth etc. Variable degrees of magnificationmay be provided, based upon detected distance from a surface beingviewed, for example.

In terms of position sensing, the preferred embodiment uses an infraredCCD (charge-coupled device) camera, preferably with a wide-angle lens206 shown in FIG. 6A. The user 202 generates a thermal image 204, whichis focused onto camera array 210 as a spot or group of pixels, allowingthe system 220 to know where a person is with no moving parts. Processor106 can then cause the perspective, depth perception, or othercharacteristics to change accordingly. If no tilt or zoom functions areprovided, a linear sensor may be substituted for a 2D sensor.

As an alternative to a fixed camera with sufficiently wide-angle lens, apanning camera may be used, as shown in FIG. 6B. Here the camera 230 istrained on a subject and, as that subject moves positional informationis sensed by pan mount 234 and communicated to alter screen 106 throughelectronics unit 240. Again, if tilt/zoom functions are provided, thecamera 230 may use tilt and/or auto focus to determine other positionalaspects of the viewer.

The invention may handle multiple viewers in different ways. Thesesolutions include (1) favoring clusters of potential viewers oversingular viewers; (2) favoring moving viewers over stationary viewers;and (3) favoring viewers actually looking at the screen over thoselooking away. One advantage of the sensor system of FIG. 6A is thatclustering is naturally accommodated. In FIG. 6C, three persons 242,244, 246 generate a composite thermal field 240, which generates arelatively large imprint 251 on sensor 210. In contrast, the narrowerthermal field 250 of single person 248 results in a smaller spot 241 onarray 210 such that, in this embodiment of the invention, theperspective of persons in the group would be favored.

The approach of FIG. 6A also naturally addresses the favoring of movingviewers over stationary viewers. Referring again to FIG. 6C, if thegroup consisting of persons 242, 244, 246 were stationary, it would be astraightforward processing task to detect that an individual is moving.In this embodiment of the invention, the moving individual 248 would bepreferred over the stationary group.

In all embodiments of the invention, sensing in the visible region ofthe spectrum may be used instead of—or in concert with—IR sensing. Thispresents advantages and disadvantages. In terms of advantages, visiblelight sensing may allow a single camera (or cameras) to detect the imageand position of the viewer (as in the mirror embodiments). Anotheradvantage is that recognition techniques may be used to determine if aparticular person is actually looking at the display in which case thatperson may be favored over individuals looking away. If multiple personsare looking at the display, other techniques such as clustering andmotion favoring may also be used. Perhaps the only disadvantage is thatimage recognition and other operations require additional processingpower, however, that is easily accommodated with modern processors.

In each case, specialized graphics processing provides the management ofthe graphics planes and any audio material, while processing rules (forexample—“take image modification instructions from the position of theclosest viewer only”) ensure that the system will not be misdirected bymovement of viewers that are on the opposite side of the room. Wheremultiple display systems are in use (as for simulating an array of“structural” windows), an overall system for management of the displaysis utilized, thus providing an integrated, coordinated system of imagingdisplays. For example—an overall image, larger than the entire displaysystem, may be utilized, or alternative schemes, in which image planesor other data may “flow” from one display screen to another displayscreen next to it.

While described herein with reference to flat-panel displays (LED, LCD,Plasma, etc.) the principles disclosed may be applied with suitableresults to any number of display technologies currently available or indevelopment today (CRT-type, front or rear projection,Electroluminescence, OLEDs, etc.). Furthermore, adjustments may beapplied to the image data to correct for any geometric distortionsintroduced due to the position of the camera(s) or display unit(s). Inaddition, alternative embodiments may utilize additional graphics planesto enhance the effect of the display.

In all cases, audio may be included, representing material that may ormay not be related to the video images presented on the screen. Thus,the system can serve the function of an enhanced video display terminal,a television viewing screen, a security monitoring system, a videoentertainment system, or any other system for which display of graphicsmaterial is of value to the viewer.

With further regard to audio, if the system is provided with stereo orsurround sound, the sound reproduction may be altered as a function withuser position, with or without a change in visual perspective. Forexample, as an individual walks past the display screen the sound ofelements in the scene (i.e., birds, vehicles, etc.) may be variedwhether or not the individual is looking at the screen. If the usermoves toward the screen, sounds may be enhanced or attenuated. Forexample, if a viewer moves toward a frog or a bird in the scene, thesounds of that creature may be enhanced, or diminished as the user movesaway.

We claim:
 1. A perspective-altering display system, comprising: a plurality of display screens displaying different portions of the same scene; a sensor for detecting the movement of a viewer relative to the display screens; at least one memory for separately storing foreground and background imagery associated with the scene; at least one processor interconnected to the memory and the sensor, the processor being operative to receive a signal from the sensor and shift the relative position of the foreground and background imagery in the displayed scene as a function of viewer movement; and wherein the foreground imagery is shifted by way of a translation to a greater extent than the background imagery as a viewer moves from side to side relative to the display screens.
 2. The display system of claim 1, wherein: the scene is larger than the plurality of display screens are capable of displaying such that the at least one memory stores non-visible foreground or background imagery; and at least some of the non-visible foreground or background imagery to becomes visible as a result of the shifting in the relative position of the foreground and background imagery in the displayed scene as a function of viewer movement.
 3. The display system of claim 1, wherein the shifting of the relative position of the foreground and background imagery in the displayed scene as a function of viewer movement causes at least some of the foreground or background imagery on one display screen to become visible on another one of the display screens.
 4. The display system of claim 1, including prerecorded foreground or background imagery.
 5. The display system of claim 1, wherein at least some of the foreground or background imagery includes imagery received through a transmission medium.
 6. The display system of claim 1, wherein at least some of the foreground or background imagery includes imagery received from a camera.
 7. The display system of claim 1, further including: a camera that records the scene through panning at sequential angles; and wherein the memory stores imagery obtained at the sequential angles for later recall as a function of user movement.
 8. The display system of claim 7, further including: an interpolator for filling in visual gaps in the scene.
 9. The display system of claim 1, wherein: the display screens are mounted on a wall having a backside; and further including a camera mounted on the backside of the wall which pans as a viewer moves, thereby imaging the scene on the display screens as virtual windows through the wall.
 10. The display system of claim 1, further including: a camera with a zoom capability for recording the scene; wherein the sensor is operative to detect the viewer's distance from the display screens; and wherein the processor is further operative to zoom in the camera as the viewer moves closer to the display screens and zoom out the camera as the viewer moves away from the display screens.
 11. The display system of claim 1, wherein: the sensor is operative to detect the viewer's distance from the display screens; and the processor is operative to increase the resolution of the scene as the viewer moves closer to the display screens and decrease the resolution of the scene as the viewer moves away from the display screens.
 12. The display system of claim 1, further including: a camera with a tilt capability for recording the scene; the sensor being operative to detect the viewer's up/down movement relative to the display screens; and wherein the processor is further operative to tilt the camera in response to the viewer's up/down movement.
 13. The display system of claim 1, further including: a camera for recording the scene, the camera having a field of view including a viewer of the display screen; and wherein the processor is further operative to flip the scene horizontally such that the display functions as a virtual mirror.
 14. The display system of claim 1, further including: a plurality of cameras for recording the scene; and wherein the processor is further operative to construct a three-dimensional image for display on the screens.
 15. The display system of claim 1, further including: a plurality of cameras for recording the scene; the processor being further operative to construct a three-dimensional image for display on the screens; and a user control a viewer to select a specific camera or cameras to see how others would view the user from different perspectives.
 16. The display system of claim 1, wherein: the sensor is a camera having an image sensor; and an image of a viewer is focused onto the image sensor as a spot or group of pixels, allowing the movements of the viewer to be tracked with no moving parts.
 17. The display system of claim 1, wherein: the sensor is a camera having at least a pan mount that tracks the movement of a viewer; and the processor is operative to shift the relative position of the foreground and background imagery as a function of the tracking.
 18. The display system of claim 1, wherein: the sensor is operative to sense a whether an individual is looking toward the display screens; and the processor is operative to shift the relative position of the foreground and background imagery by favoring individuals looking toward the display screens.
 19. The display of claim 1, wherein the display screens are contiguous. he display of claim 1, wherein the display screens are not contiguous.
 20. The display of claim 1, wherein the display screens are not contiguous. 